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Information  must  be  sorted  out  and  fused  not  only  to  allow  commanders  to  make 
situation  assessments,  but  also  to  support  the  generation  of  hypotheses  about  enemy  force 
disposition  and  enemy  intent.  While  incredible  progress  has  been  made  on  information 
fusion  in  recent  years,  the  sensing  capabilities  available  have  advanced  even  faster. 
Moreover,  the  sheer  number  of  platforms  available  to  commanders  has  dramatically 
increased.  Perhaps  more  importantly,  the  context  in  which  information  is  being  fused, 
shared  and  used  is  changing  dramatically  from  carefully  defined  hierarchical  information 
pathways  to  a  dynamic,  open  net-centric  information  sharing  and  planning.  In  this  work, 
we  focused  on  issues  related  to  this  new  reality. 

Our  research  has  focused  on  the  problems  of  developing  integrated  techniques  for  fusion 
levels  2  and  4.  Our  work  can  be  roughly  divided  into  modeling,  understanding  and 
improving  sensor  fusion  in  a  net-centric  environment  (level  2)  and  modeling, 
understanding  and  improving  humans  ability  to  use  multiple  mobile  sensor  platforms  to 
collect  information  required  for  planning  and  acting. 

In  this  report  we  present  a  very  brief  summary  of  our  work  in  each  of  these  areas 
accompanied  by  reprints  of  papers  presenting  the  research  in  detail. 

Level  2:  techniques  for  integrating  diverse  sensors  and  recognizing 
aggregated  forces 
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One  of  the  emerging  problems  in  Network  Centric  Warfare  is  where  to  fuse  data  and 
make  decisions  in  a  large  distributed  system.  We  have  been  looking  developing  and 
analyzing  models  of  how  information  might  be  shared  and  fused  in  very  large 
cooperative  groups  [1 , 2, 3, 4,  6].  A  key  finding  in  that  work  is  that  the  details  of  the 
network  structure  and  how  conclusions  are  shared  have  a  very  large  impact  on  overall 
performance.  In  [1],  we  present  a  completely  local  algorithm  that  helps  teams  reach  more 
accurate  conclusions  more  regularly.  We  added  simple  models  of  human  cognitive 
processes  to  these  models  and  looked  at  how  human  characteristics  might  influence  large- 
scale  fusion  [5].  Finally,  we  began  to  look  at  lightweight  algorithms  that  could  be  used  to 
effectively  share  information  in  large  networks,  while  respecting  bandwidth  constraints 
[7,  8], 

Level  4:  methods  for  tasking  assets  or  assisting  humans  to  acquire  new 
information 

The  process  of  acquiring  information  for  fusion  or  acting  on  it  after  it  is  acquired  requires 
tight  coordination  among  sensor  platforms  and  other  assets.  As  the  size  of  such  systems 
increases  it  becomes  necessary  to  automate  this  coordination.  Conventional  approaches 
to  automated  coordination  do  not  scale  well  above  10-20  entities  and  become  very 
difficult  for  humans  to  understand  or  control  at  even  smaller  numbers.  Much  of  our  work 
under  this  contract  has  focused  on  how  humans  cm  effectively  manage  an  increased 
number  of  mobile  sensor  platforms.  One  thread  of  research  was  related  to  high-level 
interface  concepts  for  making  sensor  data  available  to  humans,  e.g.,  by  abstracting  away 
from  individual  sensors  [10]  or  changing  time  constraints  on  how  humans  must  deal  with 
data  [11, 12, 15].  Another  research  thread  looked  at  how  humans  and  robots  were 
organized  to  manage  the  data  volume.  One  paper  looked  at  how  human  organizations 
could  be  organized  better  [9],  another  looked  at  humans  as  a  sensor  to  be  leveraged  by  a 
robot  team  [13],  others  looked  at  how  teams  of  humans  would  manage  teams  of  robots 
[23, 24].  A  series  of  important  experiments  with  real  human  subjects  looked  in  depth  at 
how  humans  can  deal  with  a  larger  number  of  robot  platforms  [14, 15,  16, 17, 18,  19, 20, 
21, 25].  These  experiments  resulted  in  new  knowledge  about  limits  on  and  factors 
effecting  humans.  Finally,  some  work  looked  at  better  autonomous  algorithms  for 
supporting  information  collection  [27, 28]. 

Simulation  Tools  and  Environments 

The  USARSim  simulation,  developed  as  a  part  of  this  research,  has  become  an  important 
tool  for  the  research  community  at  large  to  look  at  issues  of  human  control  of  multiple 
robots  [22].  In  collaboration  with  NIST,  we  have  continued  to  developed  the  simulator. 
Urban  Search  and  Rescue  Simulator  (USARSim)  is  now  the  platform  for  the  search  and 
rescue  competition  at  RoboCup,  a  competition  that  attracts  10s  of  teams  each  year. 
USARSim  nowr  uses  a  newer  version  of  the  Unreal  Tournament  (UT)  engine,  UTS,  and 
has  been  made  distributed.  This  upgrade  allows  better  physics,  better  graphics  and  many 
robots  in  the  same  environment  Our  work  with  human  control  of  wide  area  search 
munitions  through  cooperative  control  software  based  on  our  work  was  used  by  Wright 
Patterson  AFB  for  a  detailed  examination  of  humans  ability  to  control  up  to  16 
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munitions,  a  study  which  definitely  showed  the  value  of  autonomous  coordination. 

Scientific  Accomplishments 

The  scientific  accomplishments  of  this  work  can  be  roughly  divided  into  two  areas:  large- 
scale  information  sharing  and  fusion  and  human  interaction  with  mobile  sensors. 

Important  scientific  advances  were  made  on  how  information  sharing  and  fusion  might 
work  in  future  net-centric  organizations.  Extensive  use  was  made  of  multi-agent 
simulation  to  understand  key  dynamic  processes  and  emergent  effects.  The  dynamics  of 
the  information  sharing  were  found  to  be  dominated  by  cascades  of  changes  in  belief  due 
to  a  single  additional  sensor  reading.  A  key  finding  was  that  large  organizations  may 
reach  the  wrong  conclusion  more  than  the  quality  of  the  incoming  sensor  information 
would  suggest.  An  algorithm  to  improve  the  ability  of  the  organization  to  reach  correct 
conclusions  was  developed.  Mathematical  models  of  the  interesting  phenomena  were 
developed,  borrowing  techniques  from  physics.  These  models  exposed  additional 
phenomena  that  simulation  could  not.  Other  fundamental  research  on  information 
propagation  in  networks  showed  that  often  random  information  propagation  was  as  much 
as  half  as  good  for  the  team  as  a  theoretical  upper  bound  on  information  sharing 
performance. 


Previous  work  had  shown  that  a  human  can  effectively  manage  up  to  between  2  and  12 
autonomous  assets  before  becoming  overloaded.  However,  relatively  little  work  had 
looked  at  wrhat  techniques  might  increase  that  limit.  Important  scientific  advances  were 
made  to  understand  the  reasons  for  the  limit  and  how  it  might  be  raised.  One  important 
finding  was  that  human  cognitive  resources  were  heavily  consumed  with  path  planning 
and  that  by  automating  that  path-planning  a  single  human  could  control  many  more 
assets.  A  much  deeper  understanding  of  how  much  time  humans  spend  monitoring  robot 
sensor  feeds  resulted  from  large  sets  of  experiments.  This  understanding  was  leveraged 
into  new  interface  concepts  for  which  preliminary  experiments  were  performed.  One  of 
these  concepts,  a  call  center,  makes  dramatically  better  use  of  human  time  by  having  the 
robots  autonomously  decide  when  they  need  human  input. 

PUBLICATIONS 

Level  2:  techniques  for  integrating  diverse  sensors  and  recognizing 
aggregated  forces 


[1]  Robin  Glinton,  Praveen  Paruchuri,  Paul  Scerri,  Katia  Sycara  “Self- 
Organized  Criticality  of  Belief  Propagation  in  Large  Heterogeneous  Teams’ ,  In 
Dynamics  of  Information  Systems:  Theory  and  Applications,  Springer  2010 

[2]  Robin  T.  Glinton,  Paul  Scerri,  Katia  Sycara  “Towards  the  Understanding  of 
Information  Dynamics  in  Large  Scale  Networked  Systems* ,  In  Proceedings  of 
FUSION  2009 

[3]  Robin  Glinton,  Katia  Sycara,  David  Scerri,  Paul  Scerri  “The  Statistical 


Information  Fusion  for  Command  and  Control  AFOSR  FA9550-07-1-0039  Final  Report 


Mechanics  of  Belief  Sharing  in  Multi  Agent  Systems"  In  Journal  of  Information 
Fusion,  Special  Issue  on  Agent-Based  Fusion.  2009. 

[4]  Praveen  Paruchuri,  Robin  Glinton,  Kaiia  Sycara  and  Paul  Scerri  “Effect  of 
Humans  on  Belief  Propagation  in  Large  Heterogeneous  Teams”  In  Dynamics  of 
Information  Systems:  Theory  and  Applications,  Springer  2010 

[5]  Paul  Scerri,  Prasanna  Velagapudi  and  Katia  Sycara  ‘Analyzing  the 
Theoretical  Performance  of  Information  Sharing’  In  Dynamics  of  Information 
Systems:  Theory  and  Applications,  Springer  2010 

[6]  Prasanna  Velagapudi,  Oleg  Prokopyev,  Katia  Sycara,  Paul  Scerri 
“Analyzing  the  Performance  of  Randomized  Information  Sharing’  In  Proceedings 
of  AAMAS09. 

[7]  Prasanna  Velagapudi,  Oleg  Prokopyev,  Paul  Scerri,  and  Katia  Sycara  “A 
Token-Based  Approach  to  Sharing  Beliefs  in  a  Large  Multiagent  Team’  In 
Control  and  Information  Systems.  Springer,  2009. 

[8]  Scerri,  P.,  Owens,  S.,  Yu,  B.,  Sycara.,  K.  “A  decentralized  approach  to  space 
deconfliction’.  In  Proceedings  of  the  Tenth  International  Conference  on 
Information  Fusion,  Quebec  City,  CA,  July  9-12,2007. 

[9]  Velagapudi,  P,  Prokopyev,  O.,  Sycara,  K.,  Scerri,  P..  ’Maintaining  shared 
belief  in  a  large  multiagent  team".  In  Proceedings  of  the  Tenth  International 
Conference  on  Information  Fusion,  Quebec  City,  CA,  July  9-12,  2007. 

[10]  Glinton,  R.,  Scerri,  P.,  Scerri,  D.,  and  Sycara,  K.  “An  Analysis  and  Design 
Methodology  for  Belief  Sharing  in  Large  Groups” ,  In  Proceedings  of  the  Tenth 
International  Conference  on  Information  Fusion,  Quebec  City,  CA,  July  9-12, 
2007 

[11]  Scerri,  P.,  Glinton,  R.,  Owens,  S.,  Scerri,  D„  and  Sycara,  K., 
“Geolocation  of  RF  Emitters  by  Many  UAVs",  In  AIAA  lnfotech@Aerospace  2007 
Conference  and  Exhibit,  2007 

[12]  Scerri,  P.,  Glinton,  R.,  Owens,  S.,  Okamoto,  S.  Sycara,  K.  "Locating  RF 
Emitters  with  Large  UAV  Teams”,  In  Proceedings  of  Autonomous  Agents  and 
Multi-Agent  Systems,  Honolulu,  Hawaii,  May  12-19,  2007. 

[13]  P.  Seem,  S  Owens,  R  Glinton,  and  K.  Sycara.  “Synergistic  integration  of 
agent  technologies  for  military  simulation".  In  KIMAS  07:  Integration  of 
Knowledge  Intensive  Multi-Agent  Systems,  Waltham,  Mass,  April  29-May  3, 
2007. 


Level  4:  methods  for  tasking  assets  or  assisting  humans  to  acquire  new 
information 

[14]  Steven  Okamoto,  Paul  Scerri,  Katia  Sycara  "The  Impact  of  Vertical 
Specialization  on  Hierarchical  Multi-Agent  Systems’ ,  In  Proceedings  of 
AAAI’2008. 


Information  Fusion  for  Command  and  Control  AFOSR  FA9550-07-1-0039  Final  Report 


[15]  Sean  Owens,  Katia  Sycara,  Paul  Scerri  “Using  Immersive  3D  Terrain  Models 
For  Fusion  Of  UAV  Surveillance  Imagery”  In  Proceedings  of  AIAA  2009. 

[16]  Prasanna  Velagapudi,  Jijun  Wang,  Huadong  Wang,  Paul  Scerri,  Michael 
Lewis,  Katia  Sycara  “Synchronous  vs.  Asynchronous  Video  in  Multi-Robot 
Search"  In  Proceedings  of  ACHI  2008. 

[17]  Velagapudi,  P.,  Wang,  H.,  Scerri,  P.,  Lewis.  M.  &  Katia  Sycara 
(2009).  Scaling  effects  for  streaming  video  vs.  static  panorama  in  multirobot 
search,  2009  International  Conference  on  Intelligent  Robots  and  Systems  (IROS 
09),  IEEE 

[18]  Lewis,  M.,  Wang,  H,  Velagapudi,  P.,  Scerri,  P.  &  Sycara,  K.  (2009).  Using 
humans  as  sensors  in  robotic  search,  Proceedings  of  the  12th  International 
Conference  on  Information  Fusion.  July  6-9. 

[19]  Prasanna  Velagapudi,  Paul  Scerri,  Katia  Sycara,  Huadong  Wang,  Michael 
Lewis,  Jijun  Wang  “Scaling  Effects  in  Multi-robot  Control”  In  Proceedings  of 
IROS  2008 

[20]  Velagapudi,  P.,  Wang,  H.,  Scerri,  P.,  Lewis,  M„  and  Sycara,  K.  (2008). 
Potential  scaling  effects  for  asynchronous  video  in  multirobot 

search,  Performance  Metrics  for  Intelligent  Systems  (PerMIS  08). 

[21]  Wang,  H.,  Lewis,  M.,  Chien,  S.  &  Velagapudi,  P.  (2009).  Scaling  effects  for 
synchronous  vs.  asynchronous  video  in  multi-robot  search,  Proceedings  of  the 
Human  Factors  and  Ergonomics  Society  53rd  Annual  Meeting  (HFES  09),  364- 
368. 

[22]  Wang,  H.,  Lewis,  M.,  Velagapudi,  P.,  Scerri,  P.,  and  Sycara  K.  (2009).  How 
Search  and  its  Subtasks  Scale  in  N  Robots,  Proceedings  of  the  Forth  ACM/IEEE 
International  Conference  on  Human-Robot  Interaction  (HRI’09),  IEEE,  141-147. 

[23]  Wang,  J.,  Wang,  H.,  Lewis,  M.,  Scerri,  P.,  Velagapudi,  P.,  and  Sycara 
K.(2008).  Experiments  in  Coordination  Demand  for  MultiRobot 

Systems,  Proceedings  of  IEEE  SMC  International  Conference  on  Distributed 
Human-Machine  Systems  (DHMS  08),  IEEE 

[24]  Lewis,  M.  &  Wang,  J.  (2009).  Measuring  coordination  demand  in  multirobot 
teams,  Proceedings  of  the  Human  Factors  and  Ergonomics  Society  53rd  Annual 
Meeting  (HFES  09).  779-783. 

[25]  Lewis,  M.  and  Wang,  J.  (2009).  Assessing  coordination  demand  in 
cooperating  robots,  In  Madhavan,  Tunstel  and  Messina  (Ed.)  Performance 
Evaluation  and  Benchmarking  of  Intelligent  Systems,  Springer,  New  York,  169- 
186. 


[26]  Lewis,  M.  and  Wang,  J.  (2008).Assessing  measures  of  coordination  demand 
based  interaction  durations,  Performance  Metrics  for  Intelligent  Systems  (PerMIS 
08). 


Information  Fusion  for  Command  and  Control  AFOSR  FA9550-07- 1-0039  Final  Report 


[27]  Lewis,  M.  (2009).  USARSim  and  HRI:  from  teleoperated  cars  to  high  fidelity 
teams,  IROS  09  Workshop  on  Robots.  Games,  and  Research:  Success  stories  in 
USARSim,  18-22 

[28]  Wang,  H.,  Chien,  S.,  Lewis,  M.,  Velagapudi,  P.,  Scerri,  P.  &  Sycara,  K. 
(2009).  Human  teams  for  large  scale  multirobot  control,  Proceedings  of  the  2009 
IEEE  International  Conference  on  Systems,  Man,  and  Cybernetics,  1306-1311. 

[29]  Wang,  J.,  Wang,  H.,  and  Lewis,  M.  (2008).  Assessing  Cooperation  in  Human 
Control  of  Heterogeneous  Robots,  Proceedings  of  the  Third  ACM/IEEE 
International  Conference  on  Human-Robot  Interaction  (HRI  08),  IEEE 

[30]  Prasanna  Velagapudi,  Sean  Owens,  Paul  Scerri  and  Katia  Sycara,  Michael 
Lewis  “Environmental  Factors  Affecting  Situation  Awareness  in  Unmanned 
Aerial  Vehicles”  In  Proceedings  of  AIAA  2009. 

[31]  Giuseppe  P.  Settembre,  Paul  Scerri,  Alessandro  Farinelli,  Katia  Sycara, 
Daniele  Nardi  “A  Decentralized  Approach  to  Cooperative  Situation  Assessment 
in  Multi-Robot  Systems"  In  Proceedings  of  AAMAS  2008. 

[32]  Ma,  Z.,  Lee,  P.,  Xu,  Y.,  Lewis,  M.  &  Seem,  P.  (2009).  An  initial  evaluation  of 
approaches  to  building  entry  for  large  robot  teams,  2009  IEEE  International 
Workshop  on  Safety,  Security  and  Rescue  Robotics  (SSRR  09),  IEEE 

[33]  Andrea  Simonetto.  Paul  Scerri,  Katia  Sycara  “A  Mobile  Network  for  Mobile 
Sensors”  In  Proceedings  of  FUSION  2008 


REPORT  DOCUMENTATION  PAGE 

Form  Approved 

OMB  No.  0704-0188 

Public  reporting  burden  for  trws  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  nstructions.  searching  existing  data  sources,  gathering  and  mamtanmg  the 
data*  needed,  and  completing  and  reviewing  this  collection  of  Information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information,  including  suggestions  for  reducing 
this  burden  to  Department  of  Defense,  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports  (0704-0188),  1215  Jefferson  Davis  Highway.  Suite  1204,  Arlington,  VA  22202- 
4302  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  any  penalty  for  failing  to  comply  with  a  collection  of  Information  if  it  does  not  display  a  currently 
valid  OMB  control  number  PLEASE  DO  NOT  RETURN  YOUR  FORM  TO  THE  ABOVE  ADDRESS. 

1 .  REPORT  DATE  (28-2-2010)  2.  REPORT  TYPE 

Final 

3.  DATES  COVERED  (1-12-2006-30-11-2009) 

4.  TITLE  AND  SUBTITLE 

Large  Scale  Distributed  computation  for  Human  Agent  Team  Coordination  in  Dynamic  Environments 

Coordination  in  Dynamic  Environments 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

FA9550-07-1-0039 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

Katia  P.  Sycara 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f,  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Carnegie  Mellon  University  University  of  Pittsburgh 

Robotics  Institute  School  of  Information  Science 

5000  Forbes  Av.  135  N.  Bellefield  Av. 

Pittsburgh,  PA  15213  Pittsburgh,  PA.  15260 

8.  PERFORMING  ORGANIZATION  REPORT 

NUMBER 

9.  SPONSORING  /  MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

Dr.  David  Luginbuhl 

AFOSR 

875  N.  Randolph  Str. 

Suite  325 

Arlington,  VA.  222203-1768 

10.  SPONSOR/MONITOR’S  ACRONYM(S) 

11.  SPONSOR/MONITOR'S  REPORT 
NUMBER(S) 

A f£l  J2f)L3-  0 

12.  DISTRIBUTION  /  AVAILABILITY  STATEMENT 

Unrestricted  -  -/V 


13.  SUPPLEMENTARY  NOTES 


14.  ABSTRACT 

Recent  research  and  technological  advances  are  harbingers  of  scenarios  where  large  scale  crisis  response  in  civilian  and  military  domains  can  be  facilitated 
by  autonomous  sensing  and  effecting  entities.  To  make  this  a  reality  scientific  research  shortfalls  must  be  addressed,  notably  the  challenge  of  scalability. 
Current  multi-agent  coordination  techniques  do  not  scale  to  large  numbers  of  agents  Therefore,  the  goals  of  this  research  are  (1)  to  discover  ways  in  which 
large  numbers  of  autonomously  computing  entities  (including  software  agents  and  physical  robots)  must  be  coordinated  in  effective  ways,  and  (2)  ways  for 
human  operators  to  supervise  and  control  robotic  agents  In  particular,  we  want  to  develop  techniques  that  allow  large  scale  coordination  in  the  execution  of 
complex  missions  in  uncertain  and  dynamically  changing  environments.  To  achieve  this  goal  a  variety  of  key  research  issues  will  be  addressed  Specifically, 
we  will  develop  (a)  tractable  automated  coordination  algorithms,  (b)  efficient  techniques  for  fusing  information,  and  (c)  techniques  for  allowing  effective  and 
flexible  control  of  agent  groups  by  small  numbers  of  human  operators. 


15.  SUBJECT  TERMS 

Distributed  computation,  multi-agent  coordination,  human  robot  interaction 


16.  SECURITY  CLASSIFICATION  OF: 

17.  LIMITATION 

OF  ABSTRACT 

18.  NUMBER 
OF  PAGES 

19a.  NAME  OF  RESPONSIBLE  PERSON 
Dr.  Katia  P.  Sycara 

a.  REPORT 
unclassified 

b.  ABSTRACT 

unclassified 

c.  THIS  PAGE 
unclassified 

None 

239 

19b.  TELEPHONE  NUMBER  ( include  area 
code)  412-268-6825 

z0l 


Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std.  239.18 


